A Hybrid Kalman-Nonlinear Ensemble Transform Filter

A hybrid ensemble transform filter is introduced which combines a Kalman-filter update provided by the analysis scheme of the local ensemble transform Kalman filter (LETKF) and a nonlinear analysis following the nonlinear ensemble transform filter (NETF). The NETF computes an analysis ensemble from particle weights and has been shown that it can yield smaller errors than the LETKF for sufficiently large ensembles. The new hybrid filter is motivated from combining the stability of the LETKF with the nonlinear properties of the NETF to obtain improved assimilation results for smaller ensembles. The hybrid filter is domain-localized as the LETKF and can hence be applied to high-dimensional nonlinear models. Its performance depends on the choice of the hybridization weight which shifts the analysis solution between the LETKF and NETF analyses

Data assimilation for nonlinear systems with a hybrid nonlinear Kalman ensemble transform filter

Ensemble Kalman filters are widely used for data assimilation applications in the geosciences. While they are remarkably stable even with nonlinear systems, it is known that they are not optimal in this case. The alternative particle filters are fully nonlinear, but difficult to apply with high-dimensional models. To combine the strengths of both filter types, a hybrid filter is introduced that combines the local ensemble transform Kalman filter (LETKF) with the nonlinear ensemble transform filter (NETF). Three variants of the hybrid filter are formulated. The hybridization is controlled by a hybrid weight. Different hybrid weights are examined and a new adaptive approach based on the ensemble skewness and kurtosis is introduced. The different hybrid filters and the schemes to compute the hybrid weight are assessed in numerical experiments with the nonlinear Lorenz-63 and Lorenz-96 models at different degrees of nonlinearity. A hybrid variant that first applies the NETF followed by the LETKF yields the best results. For the Lorenz-96 model, error reductions by up to 21.5% compared with the LETKF are obtained for the same ensemble size. Computing the hybrid weight based on skewness and kurtosis combined with the effective sample size yields the lowest estimation errors and the overall highest stability of the hybrid filter. The new hybrid filter applies localization and inflation and is hence also usable with high-dimensional models and can potentially provide a robust way to account for leading nonlinearity with small ensembles in nonlinear data assimilation applications

Building Ensemble-Based Data Assimilation Systems with Coupled Models

Discussed is the construction of programs for efficient ensemble data assimilation systems based on a direct connection between a coupled simulation model and ensemble data assimilation software. The strategy allows us to set up a data assimilation program with high flexibility and parallel scalability with only small changes to the model. The direct connection is obtained by first extending the source code of the coupled model so that it is able to run an ensemble of model states. In addition, a filtering step is added using a combination of in-memory access and parallel communication to create an online-coupled ensemble assimilation program. The direct connection avoids the common need to stop and restart a whole coupled model system to perform the assimilation of observations in the analysis step of ensemble-based filter methods like ensemble Kalman or particle filters. Instead, the analysis step is performed in between time steps and is independent of the actual model coupler. This strategy allows us to perform both in-compartment (for weakly coupled assimilation) and cross-compartment (for strongly coupled assimilation) assimilation. The assimilation frequency can be kept flexible, so that assimilation of observations from different compartments can be performed at different time intervals. Using the parallel data assimilation framework (PDAF, http://pdaf.awi.de), the direct connection strategy will be exemplified for the ocean-atmosphere model ECHAM6-FESOM

Building a Scalable Ensemble Data Assimilation System for Coupled Models

Efficient ensemble data assimilation with coupled models poses particular challenges due to the comp lexity of the model system and due to its high computational cost. On the methodological side, one h as to account for different time scales, but also distinct correlation lengths, of different model c ompartments like the ocean and the atmosphere. Computationally, one often has to deal with multiple program executables, a coupler software, observation handling for different model compartments, and a large number of processors required to compute a complex coupled model. This contribution focuses on the computational aspects. Discussed are the steps required to build a highly scalable and flexible data assimilation system can be built on the basis of the Parallel Data Assimilation Framework (PDAF, http://pdaf.awi.de) using the example of the coupled climate model AW I-CM (Sidorenko et al., Climate Dynamics, 44 (2015) 757-780). AWI-CM consists of the finite-element sea ice-ocean model FESOM, which uses an unstructured model grid, and the model ECHAM6 for the atmosphere. The model coupling is implemented with OASIS-MCT and the model system consists of two separate executable programs for the ocean and atmosphere. Next to the implementation steps, the scalability of the assimilation system is discussed with a realistic configuration of AWI-CM. The high scalability is obtained by an online-connection strategy for the data assimilation system. First, the parallelization of the coupled model system is modified so that the coupled model can perform ensemble forecasts. Second, the analysis (solver) step is directly inserted into the time-stepping loops of each model compartment. Augmenting the coupled model in this online way, the ensemble information is kept in memory and transferred by parallel communication when necessary. Thus, one avoids the need to repeatedly write an ensemble of model fields into files and read them again for the analysis step. Further, the coupled model is only started once and there is no need to stop and restart the whole coupled model to compute the analysis step. Instead, the analysis step is performed in between time steps and is independent of the actual model coupler. These modifications of the model are supported by the framework structure of PDAF. In addition to the parallel online connection for the data assimilation system, the analysis step has to be parallelized. Here, the different model compartments are treated like parallel subdomains of the model. In this way, one can one can use the data assimilation algorithms provided by PDAF and can implement and perform the analysis step in analogy to uncoupled models. However, one has to take into account the different model grids and possible distinct ways in which the model compartments store their model fields. This results in a data assimilation system that can perform the assimilations both in-compartment (for weakly coupled assimilation) and cross-compartment (for strongly coupled assimilation)

Building Ensemble-Based Data Assimilation Systems for High-Dimensional Models

Different strategies for implementing ensemble-based data assimilation systems are discussed. Ensemble filters like ensemble Kalman filters and particle filters can be implemented so that they are nearly independent from the model into which they assimilate observations. This allows to develop implementations that clearly separate the data assimilation algorithm from the numerical model. For coupling the model with a data assimilation software one possibility is to use disk files to exchange the model state information between model and ensemble data assimilation methods. This offline coupling does not require changes in the model code, except for a possible component to simulate model error during the ensemble integration. However, using disk files can be inefficient, in particular when the time for the model integrations is not significantly larger than the time to restart the model for each ensemble member and to read and write the ensemble state information with the data assimilation program. In contrast, an online coupling strategy can be computational much more efficient. In this coupling strategy, subroutine calls for the data assimilation are directly inserted into the source code of an existing numerical model and augment the numerical model to become a data assimilative model. This strategy avoids model restarts as well as excessive writing of ensemble information into disk files. To allow for ensemble integrations, one of the subroutines modifies the parallelization of the model or adds one, if a model is not already parallelized. Then, the data assimilation can be performed efficiently using parallel computers. As the required modifications to the model code are very limited, this strategy allows one to quickly extent a model to a data assimilation system. In particular, the numerics of a model do not need to be changed and the model itself does not need to be a subroutine. The online coupling shows an excellent computational scalability on supercomputers and is well suited for high-dimensional numerical models. Further, a clear separation of the model and data assimilation components allows to continue the development of both components separately. Thus, new data assimilation methods can be easily added to the data assimilation system. Using the example of the parallel data assimilation framework [PDAF, http://pdaf.awi.de] and the ocean model NEMO, it is demonstrated how the online coupling can be achieved with minimal changes to the numerical model

A Unification of Ensemble Square Root Kalman Filters

In recent years, several ensemble-based Kalman filter algorithms have been developed that have been classified as ensemble square-root Kalman filters. Parallel to this development, the SEIK (Singular ``Evolutive'' Interpolated Kalman) filter has been introduced and applied in several studies. Some publications note that the SEIK filter is an ensemble Kalman filter or even an ensemble square-root Kalman filter. This study examines the relation of the SEIK filter to ensemble square-root filters in detail. It shows that the SEIK filter is indeed an ensemble-square root Kalman filter. Furthermore, a variant of the SEIK filter, the Error Subspace Transform Kalman Filter (ESTKF), is presented that results in identical ensemble transformations to those of the Ensemble Transform Kalman Filter (ETKF) while having a slightly lower computational cost. Numerical experiments are conducted to compare the performance of three filters (SEIK, ETKF, and ESTKF) using deterministic and random ensemble transformations. The results show better performance for the ETKF and ESTKF methods over the SEIK filter as long as this filter is not applied with a symmetric square root. The findings unify the separate developments that have been performed for the SEIK filter and the other ensemble square-root Kalman filters